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Introduction to the Application of Web-Based Surveys 
By Annemarie Timmerman 



Introduction 

Although still evolving, the Internet has been used for survey research 
in a variety of ways and for different purposes (Kay and Johnson, 
1999). The Internet with its almost instantaneous transmission of data 
and communications provides unlimited opportunities for researchers 
conducting survey research. While a breadth of literature has been 
written on the methodological concerns with electronic surveys 
(Manfreda, Batagelj, and Vehova, 2002) standards to control for 
sampling error, user-response error and measurement error in online 
survey research are still being decided. As the use of online survey 
research grows and as commercial online survey products are 
developed deciphering the drawbacks and the assets in web-based 
surveys becomes increasingly important. 

Over the last decade, electronic surveys have evolved from disk-by- 
mail surveys, to e-mails with embedded or attached surveys and 
finally to web-based surveys posted on the Internet (Shannon, et. al . , 
2002). With web-based surveys, participants are usually notified by e- 
mail to participate in the survey. The e-mail generally includes a link 
to the URL (uniform resource locator) web address of the survey. The 
focus of this article is on web-based surveys. 

Unique Qualities 

As a survey tool, web-based surveys have qualities similar to print 
surveys, interview surveys and telephone surveys, as well as qualities 
unique only to web-based tools. Like print surveys, web-based surveys 
are visual in nature. However, increased manipulation of font size, 
color, layout, graphics, and animation expand the visual language of 
web-based surveys over print (Cooper, 2000). 

Like paper surveys and IVR (individual voice response) telephone 
surveys, web-based surveys are self-administered questionnaires. 
Redline and Dillman (1999) present the complexity of skip-pattern 
designs and respondent navigation techniques offered in self- 
administered surveys. Web-based surveys and online survey tools 
offer a host of skip-pattern designs that can exasperate or simplify the 
navigation through a survey for the respondent depending on the 
quality of the technical design of the web-based survey (Dillman, 
Tortura, and Bowker, 1998). 




In this paper, we discuss some basic assumptions and issues 
concerning web-based surveys. Several assumptions regarding the 
advantages of web-based surveys will be addressed. Issues pertaining 
to web-based surveys will center on sampling error, measurement 
error, and no response error. Finally, we will look at some critical 
aspects of web-based survey design and recommendations for online 
survey tools. 

Assumptions 

Cost 

Lower costs are often presented as one of the benefits of Internet 
surveys (Dillman, 2000). However, the cost benefit of online surveys 
versus paper surveys is dependent on which items are included in the 
cost analysis. In an analysis of the cost of paper surveys vs. web- 
based surveys in a Students Life Experiences Survey study, Schaefer 
determined that the cost of disseminating a paper survey was $2.07 
per student while the cost of the web-based survey was $.88 per 
student (Gunn, 2002). While Schaefer's analysis of the costs of the 
paper survey versus the web-based survey are certainly accurate, the 
population surveyed needs to be considered. University students have 
greater access to computers and the Internet than the general 
population. The technology infrastructure and online support network 
that exists in a university environment does not extend to other 
populations, specifically in rural, low-income populations. Generally, 
the Internet access and the use of the university's infrastructure are 
free to students and the faculty researchers. A study by Asch 
concluded that web-based surveys appear to be economical when they 
are the only mode of response with postal mail used only as a follow- 
up to non-respondents and over 580 responses are obtained over the 
web (Schonlau, Fricker and Elliott, 2001). It would be difficult to use 
web-based surveys as the only method of data collection among rural 
populations where Internet access is less available and obtain that 
high of a response rate. Moreover, there are likely to be additional 
Internet access costs for rural populations that will prohibit a high 
response rate. 

Ease of Use 

While well-designed web-based surveys may eliminate some of the 
navigation difficulties found in print surveys, they include an additional 
set of skills unnecessary in print, interview or telephone surveys. 




Before participating in any self-administered web-based survey, 
participants need basic computer and Internet browsing skills. Lack of 
these skills hinders participants from completing web-based surveys. 
Where researchers have offered both a web-based and a print version 
of a survey, some participants choose to complete the print version 
(Zhang, 1999). Even self-reported active users of the Internet often 
opted for a postal mail survey (Zhang, 1999). Additionally, access to 
Internet technology varies within a population. Incompatibility issues 
with hardware and software, varying Internet access speeds, and the 
look and feel of different web browsers compound the frustration and 
non-response of participants uncomfortable with a web-based format 
and complicate the design and implementation of web-based surveys 
(Dillman & Bowker, 2001). 

While web-based surveys offer new opportunities and resources for 
survey research, the advantages must be measured in terms of their 
applicability with a given population. The purpose here is not to 
dispute the advantages of web-based surveys but to put the 
advantages into proper perspective. Web-based surveys are very 
different from other survey methods in their execution. Before 
assuming web-based surveys are the most advantageous method for 
surveying a target population, it should be determined if those 
advantages exist within that population. 

Disadvantages 

The disadvantages of web-based surveys concern the inability to 
compensate for four common errors of survey research: coverage 
error, sampling error, measurement error and nonresponse error ( 
Dillman & Bowker, 2001). The purpose of survey research is to 
describe what people think or do (Cooper, 2000). An accurate 
description of a population is reliant on surveying a representative 
sample and obtaining accurate responses from the sample. Salent and 
Dillman (1994) discuss four sources of error in survey research that 
result in an inaccurate description of a population. These errors are: 

• Coverage Error: The sample frame drawn from a 
population does not represent all elements of a given 
population. 

• Sampling Error: The result of surveying only one subset of 
the population rather than the entire population. 

• Measurement Error: A respondent's answer to a question is 
different from the actual answer due to poor question 
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wording, poor interviewing, survey mode effects and/or 
some aspect of the respondent's behavior. 

• Nonresponse Error: Those who did not respond to a 

questionnaire are different from those who did respond to 
the questionnaire in a way that is important to the study. 

If the non-respondents had responded, they would have 
provided different answers to the survey questions than 
those who did respond to the survey. 

Coverage Error 

Internet access is not ubiquitous. According to the September 2001 
U.S. Census Bureau's Current Population Survey, 65.6 percent of the 
U.S. population were computer users (National Telecommunications & 
Information Administration, 2002). Use of the Internet is significantly 
higher among higher educated and higher income populations 
(National Telecommunications & Information Administration, 2002). 
Furthermore, computer and Internet use rates fall significantly among 
populations older than 55 (National Telecommunications & Information 
Administration, 2002). 

Obviously, coverage of the population is a great concern when 
conducting web-based surveys. It is easy to see how not everyone 
included in a target population of a study would be represented in the 
sample frame if the Internet is the only mode of delivery of a survey. 
To accurately depict and to make inferences of a population, a 
probability sample must be representative of the target population. 
Without accurate representation, a sample frame drawn strictly from 
Internet users results in merely a convenience sample for which 
statistical inferences cannot be made about the target population. 

The aforementioned census data establishes that the proportion of a 
population that can be reached with web-based surveys is not 
universal, therefore, unlikely to be representative of all populations. 
Further, the difference in demographics of those who can participate in 
a web-based survey from those who lack the means to participate in a 
web-based survey is significant. Survey researchers who employ web- 
based surveys as a mode of data collection suggest that target 
populations must be clearly defined and authenticated before 
conducting web-based research to minimize the possibility of coverage 
error (Shannon, e. al., 2002). A representative sample of a target 
population may only be possible with a multimode survey approach 
utilizing a combination of print, telephone, interview, and web-based 
venues (Dillman, 2000). 



Sampling Error 

While coverage error refers to people missing from the sample frame 
(those without Internet or Web access in the case of web-based 
surveys), sampling error occurs during the selection of a sample from 
the population (Salent and Dillman, 2001). A sampling frame is a list 
of persons(or identifiers) from which a sample is drawn (Salent and 
Dillman, 1994). The difficulty is assuring that every member of a 
population is included in a sampling frame. For example, the 
uniformity and regional area code of telephone numbers allow a 
method of random selection to be devised for telephone surveys. 
Furthermore, telephone directories provide comprehensive lists of the 
telephone addresses of a regional population. 

E-mail is the primary and best form of contact for notifying 
participants of a web-based survey. The lack of uniformity or regional 
code in e-mail addresses prevents the development of any such 
random sampling selection. Sampling frames using listed e-mail 
addresses exclude numerous Internet users who do not appear in 
online directories. Online e-mail directories frequently rely on 
volunteer registration rather than automatic, registration prevalent 
with telephone subscriptions. As a result, no method exists where a 
representative, random sample of Internet users can be drawn from a 
diverse population. 

However, within organizations (for example a state government 
education agency or a local school district) whose entire survey 
population has Internet access and e-mail, sampling error can be 
contained. Within confined organizations, generally, a comprehensive 
list of e-mail addresses does exist online and there is a uniform code 
for assigning e-mail addresses. Conducting a census of the entire 
population rather than a sample is possible within such restricted 
populations. Sampling issues would be no greater with web-based 
surveys than print surveys within these contained populations. 

Measurement Error 

Measurement errors occur during the data collection process. 
Measurement errors in surveys are deviations of the respondent's 
answers from the actual answers (Salent and Dillman, 1994). Salent 
and Dillman (1994) list three sources of measurement errors pertinent 
to web-based surveys: the survey method, the questionnaire design, 
and the respondent. Salent and Dillman (1994) found that different 




survey methods can yield very different answers for the same 
question. The method of survey places different demands on the 
respondent. Self-directed surveys allow the respondent to control the 
pace, and direction of the completion of a survey. Moreover, the 
participant has more control over the response to a question or the 
choice to not respond to a question. 

While wording of questions can have an impact on survey response, 
the recommendations for general wording and design of questions for 
web-based surveys is no different than for print surveys. A poorly 
designed question is a poorly designed question no matter what 
medium the survey is delivered. Rea and Parker (1997, p.45 - 63) 
provide several recommendations for phrasing and formatting survey 
questions. 

Flow and design of web-based surveys and its impact on measurement 
errors has been the topic of recent studies (Dillman, Tortura, and 
Bowker, 1998; Dillman, Tortora, and Bowker, 1998; Crawford, Couper, 
and Lamias, 2001; Couper, M., Traugott, and Lamias, M., 2001). The 
appearance of a web-based survey can vary from respondent to 
respondent because of the look and feel of different web browsers, 
user preferences, and variations in hardware configurations. Design of 
Web-based surveys becomes much more important because of graphic 
tools available to the designer (color, sound, images, animation, etc.), 
the varying computer skills of respondents and the variation in 
appearance of a web-based survey from computer to computer 
(Cooper, 2001). In a study of college students' attitudes on Affirmative 
Action, Couper, Traugott, and Lamias (2001) experimented with 
different layouts and designs of a web-based survey. They found that 
students completed the survey faster and answered more questions 
with multiple-item screens. They also saw a stronger correlation 
among items on the same screen than when questions were presented 
one screen at a time. Dillman and Bowker (2001) noted that excessive 
navigational controls that require a respondent to perform several 
steps before moving to the next question lead to frustration and 
abandonment by the respondent. 

Nonresponse Error 

Nonresponse error refers to the participants in a sample unwilling or 
unable to complete the survey. Generally, web-based surveys have a 
lower response rate than mail surveys (Solomon, 2001). Several 
reasons for lack of response to web-based surveys have been offered. 
Crawford, Couper, and Lamias (2001) noted that lack of experience of 



researchers in achieving high response rates may be one factor. 
Dillman and Bowker (2001) suggest that simpler web designs that load 
quickly and easily result in higher response rates. More complicated 
web-based surveys that take longer to load and cause compatibility 
problems for older computers may result in abandonment before the 
respondent ever actually begins a survey. It is crucial that web-based 
surveys provide a format that is easy to load and is compatible with a 
variety of computer systems. Solomon (2001) presents data from a 
meta-analysis that follow-up contacts with nonrespondents and 
personalized contacts increased the response rate to web-based 
surveys. 

Web-Based Survey Design 

Previous studies have shown that the design of a web-based survey 
can affect who completes the survey (Dillman and Bowker, 2001), the 
number of questions completed within the survey (Salent and Dillman, 
1994), and the actual answers from the respondent (Couper, Traugott, 
and Lamias, 2001). With the graphic and multimedia capabilities of the 
computers, the survey researcher has an almost unlimited set of 
design choices in developing web-based surveys. As a result, the 
design and format of web-based surveys is highly variable. Clear 
standards for the design of web-based surveys are lacking. However, 
studies on the affects of certain design features on response do 
provide some guidance on survey design. 

Multi-page vs. Single-page Layout 

One of the central issues in web-based survey design is the advantage 
and disadvantage of the multi-page layout of a web-based survey 
(Dillman, 2000; Couper, Traugott, and Lamias, 2001). A multi-page 
layout refers to only a portion of the survey questions displayed on a 
web page. As the respondent completes a question, they are 
presented with another question. The respondent does not see other 
questions on a survey until the displayed question(s) is/are completed. 
Variations of multi-page layouts group same type questions or same 
topic questions on a single web page. 

A single-page design follows the more traditional pattern of print 
surveys. All the questions of the survey are on a single web page. The 
respondent can preview all of the questions at one time. The 
respondent navigates through the survey using the scroll bar. 

Dillman (2001) suggests that multi-page layouts prevents the 
respondent from gaining the overall content of the survey. Without the 
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ability to preview the entire survey before answering the questions, 
the respondent has no means for developing a sense of meaning or 
purpose of the survey. Studies also reveal that multi-page surveys 
take significantly longer to complete than single-page surveys 
(Manfreda, Batagelj, and Vehova, 2002). However, multi-page designs 
enable the researcher to create skip patterns and branching 
unobservable to the respondent (Couper, Traugott, and Lamias, 2001). 
Designed properly multi-page layouts have the potential of making a 
longer, more complicated survey easier for the respondent. Ultimately, 
the decision to construct a multi-page or a single-page layout is 
dependent on several variables: the computer skills of the 
respondents, the length of the survey, the complexity of the survey 
and if multiple survey modes are being conducted (Dillman, 2000). 

Progress Indicators 

Dillman and Barker (2001) found that one source of frustration for 
respondents in web-based surveys was not knowing how close they 
were to the end of the questionnaire. This was especially true with 
multi-page layouts where the respondent had no indication of how 
long the survey was. Recent studies have examined the use of 
progress indicators that inform the respondent how much of the 
survey they have completed. Progress indicators may take the form of 
a pie chart, or horizontal bar in one corner of the screen illustrating 
the percentage of the survey that is complete. The purpose of progress 
indicators is to motivate the respondent to complete the survey. 
Studies have revealed mixed results in higher completions rates with 
progress indicators (Crawford, Couper, and Lamias, 2001). In a study 
of college students' attitude toward affirmative action, Couper, 
Traugott, and Lamias (2001) found no significant difference in the 
number of partial response or survey abandonment between 
respondents who had a progress indicator within their web-based 
survey and respondents who had no progress indicator. Couper, et. al. 
hypothesized that the additional download time needed to complete 
surveys with the progress indicator may have counteracted with the 
affect of including the progress indicator. Other studies have also 
found that demographics and the length of the survey are a factor in 
the usefulness of a progress indicator (Crawford, Couper, and Lamias, 
2001 ). 

Layout 

The skills required to create web-based surveys are different from the 
skills to create other types of surveys (Shannon, et. al., 2002). 
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Depending on which software product, online survey tool or 
programming language used to develop a web-based survey, 
moderate to expert skills are required. Additionally, knowledge of good 
web page design is necessary to create an interactive, easily navigated 
web survey (see Criteria For Evaluating Web Sites, Appendix A). 
Furthermore, more research and knowledge are needed on the impact 
of web design and web features on response error and measurement 
error (Dillman, 2000). Due to the technological knowledge and skill 
required to develop web-based surveys, leadership in the development 
of web-based surveys has originated from the technology professional 
rather than research professionals. The challenge for survey 
methodologists is to merge sound principles of survey design and 
implementation with sound principals of web design (Dillman, 2000; 
Dillman & Bowker, 2001). 

The most thorough attempt to provide guidelines for web-based 
survey design was done by Dillman, Tortura and Bowker (1998). 
Appendix B. lists the fourteen principles constructed and the type of 
survey error they address. The recommendation to address sampling 
error with a probability sample is to limit access to the survey by 
assigning each respondent a PIN number. Other principles addressing 
sampling and coverage errors are constructing web questionnaires that 
scroll from question to question and providing an indicator of where 
the respondent is in the survey. Most of the principles aim at reducing 
frustration of the respondent with hardware incompatibility and slow 
data access speeds and reducing nonresponse error. Overall, these 
principles encourage simplistic design of surveys. Dillman, Tortura and 
Bowker (1998) readily admit that these principles were based mostly 
on observation and that empirical studies are still needed to verify and 
improve upon this list. 

Conclusion 

The research on web-based surveys to date indicates that there are 
many possibilities for the use of web-based surveys in methodological 
research. More empirical studies need to be conducted that provide 
recommendations for developing web-based surveys that control for 
coverage error, sampling error, measurement error and nonresponse 
error. Based on the advice from numerous research professionals, 
web-based surveys are best employed with well-defined, targeted 
populations where Internet access is widely available (Shannon, e. al., 
2002 ). 
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With the numerous survey development products now available, the 
construction of web-based surveys is dramatically easier to construct 
and deploy. However, the development of survey software and online 
survey products make the need for empirically sound design 
recommendations more critical. Based on the research thus far on 
web-based surveys, the following features are recommended for online 
survey products: 

1. Provide the option to include a progress indicator on the survey. 

2. Provide a method for respondents to print a printer-friendly 
version of the survey. 

3. Provide the researcher flexibility in the design and formatting of 
the survey. 

4. Provide methods of authentication for probability samples such 
as password protection or PIN numbers. 
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Appendix A. CRITERIA FOR EVALUATING WEB SITES 
(Southern Regional Education Board) 

Content 

Accuracy: 

Error-free information 
Current information 
Updated frequently 
Recent "last" update 

Objective, balanced presentation of information 
Bias-free viewpoints and images 

Correct use of grammar, spelling, and sentence structure 

Primary outlink (link that takes you to additional site) content is relevant, 
authentic, and appropriate 

Authority: 

Expertise/reputation of author/designer 
Contact information for author/designer 
Expertise/reputation of host site 

Appropriateness: 

Concepts and vocabulary relevant to students' abilities 
Information relevant to the North Carolina K-12 curriculum 

Interaction compatible with the physical and intellectual maturity of intended 
audience 
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Scope: 



Information of sufficient scope to adequately cover the topic for the intended 
audience 

Logical progression of topics within original site (site being evaluated) and 
primary outlinks 

Information offered not easily available in other sources 
Technical Aspects: 

Navigation: 

Ready access to site; site not overloaded 
Images load within reasonable timeframe 

Intuitive icons, menus, and directional symbols that foster independent use 

Inlinks (links that take you to locations within the original site) that allow easy 
navigation throughout the site 

Standard multimedia formats 

Logical options for printing/downloading all or selected text and graphics 

Presentation: 

Site follows good graphic design principles 

Screen displays uncluttered and concise 

Captions, labels, or legends for all visuals 

Legible text and print size appropriate for the intended audience 

Graphics and art functional, not merely decorative 

Information presented through text, motion, still images, and sound 

Information presented in a manner to stimulate imagination and curiosity 

Product advertising not intrusive 
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Appendix B: Principles for the design of web surveys (Dillman, Tortura and Bowker, 
1998) 



Principle 


Sampling 


Coverage 


Measurement 


Non-Response 


1. Introduce the web questionnaire with a 
welcome screen that is motivational, 
emphasizes the ease of responding, and 
instructs respondents on the action needed 
for proceeding to the next page. 








X 


2. Provide a PIN number for limiting 
access only to people in the sample. 


X 


X 






3. Choose for the first question an item 
that is likely to be interesting to most 
respondents, easily answered, and fully 
visible on the first screen of the 
questionnaire. 








X 


4. Present each question in a conven-tional 
format similar to that normally used on 
paper self- administered questionnaires. 






X 


X 


5. Restrain the use of color so that 
figure/ground consistency and read-ability 
are maintained, navigational flow is 
unimpeded, and measurement properties of 
questions are maintained. 






X 




6. Avoid differences in the visual 
appearance of questions that result from 
different screen configurations, operating 
systems, browsers, partial screen displays 
and wrap-around text. 




X 


X 


X 


7. Provide specific instructions on how to 
take each necessary computer action for 
responding to the questionnaire and other 
necessary instructions at the point where 
they are needed. 








X 


8. Use drop-down boxes sparingly, 
consider the mode implications, and 
identify each with a "click here” 
instruction. 






X 




9. Do not require respondents to provide 
an answer to each question before being 
allowed to answer any subsequent ones. 








X 


10, Provide skip directions in a way that 
encourages marking of answers and being 
able to click to the next applicable 
question. 






X 




1 1 . Construct web questionnaires so they 
scroll from question to question unless 
order effects are a major concern, and/or 
telephone and web survey results are being 
combined. 




X 


X 


X 


12. When the number of answer choices 
exceeds the number that can be displayed 
in a single column on one screen, consider 
double-banking with an appropriate 
grouping device to link them together. 






X 




1 3. Use graphical symbols or words that 
convey a sense of where the respondent is 
in the completion process, but avoid ones 
that require significant increases in 
computer memory. 




X 




X 


14. Exercise restraint in the use of question 
structures that have known measurement 
problems on paper questionnaires, e.g.. 






X 


X 
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check-all -that-apply and open-ended 
questions. 
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